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of the S (L) £A interpretive frameworks, considered to provide useful 
and aeaningful feedback to a learner using 5(L)EA as a reflection 
aedius, is examined* (The interpretive framework examined involved 
the development and use of a self ^designed set of S (L) EA card 
clusters.) Methodology and findings are presented, with the findings 
also shown in diagrams and tables. The final two topics cover the 
evaluation research applications of S (L) EA, and experience with 
S (L) EA in a program evaluation. (SH) 
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Background to S(L)EA 

This instrument is based upon the development of another 
instrument called SHAPES - Shared Process Evaluation System, 

SHAPES is used for conmiunity development project evalua- 
tion. A numbar of assumptions , processes and instrumentation 
design factors have either been directly adapted from SHAPES or 
modified to suit the orientation and application of S(L)EA. 

An analysis on the following pages identifies the tran- 
sfer of SHAPES documentation to S(L)EA. This documentation is 
presented in chronological order* That is, there were four 
sources of data on SHAPES dating back to November 1974 and each 
of these (starting with the earliest item) has been analysed 
for assumptions ? processes and instrumentation design traits 
which have a bearing upon S(L)EA, The analysis of SHAPES docu-- 
ments is a major grounding for SCL)EA. 

In addition, however, certain literature which supports 
and helped in the conceptual development of S(L)EA are (see 
bibliography for further details) John Flannagan's "Measuring 
Human Performance"* This is the only major foundation work on 
Critical Incident Analysis that the SHAPES investigators (Davie, 
at al) and myself have found so far. 
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^Developed at the Ontario Institute for "Studies in Education 
Toronto, Canada by Dr. Lynn Davie, Richard Cawley, Terry Patterson 
and Dorothy MacKeracher, Informal papers and a manual on SHAPES 
are available through the Adult Education Department, OISE. 



ANALYSIS OF THE REAL^IONSHIP BETWEEN SHAPES AND S (L) EA 



On the following pages: 
1/ A numbering system is used for each statement quoted 

from SHAPES documents (e*g. 1,01/01) 

- the first digit represents a code found corresponding 
to a reference in the bibliography; 

- the next two digits represent page numbers in that 
reference ; 

- the number after the stroke (/) is an indeKing of the 
assumptions, processes and instrumentation design traits, 

2/ Three coding columns are as follows: 

i- Classification Code Identifies whether the statement is 
largely being construed as either 
A - Assumption 
P - Processes 
IDT - Instrumentation Design Traits, 
ii' Application Code Identifies how A, P and IDT were 
transferred to S(L)EA 

0 = an optional assumption - this can apply dependin< 
upon how S(L)EA is used, 
D/A = a direct application of A,P or IDT to S(L)EA, 
M = modifications made - see Referenca Code for 
additional information , 
N/A = not applicabe to S(L)EA, 
iii- Reference Code Only applies where a "M" or 0 appears 

in the application code column and refers to a section- at 
the end of this analysis* ^ 
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S^^E'S ASSUHPTIONS, PROCESSES Classification 


^ — ' 
Reference 


Appliea^ 


COHMENTS 


a INSTRUMENT DESIOT TRAITS tion Code 


Code 


tion Code 


; 1.01/01 shape's la intended to IDT 
"allow a display of the 
Interaetlon between 
groups a individuals," 


See 
co^enta 


0 


not the primary 
intention of 
S(L)EA 


1,01/02 "It organiaea a variety P 
of aatlvltiea into a 
coherent framework". 


See 
COTments 


D,A, 


S(L)EA is as 
much a syatem 
as an Inatrumenl 


1,01/03 Resulting from this dls- P 
play, a person can mike 
Judgementiabout the 
tlMng and nature of his 
Interventlone In order 
to faellltate positive 
outcomes 


00l//*3l#b 


H 


This originally 
came from T#rry 
Patterson's 
paper see 
Ref* "Temporal 
$£ Spacialg 
Relations , 


1*01/04 "Huaan activity related to A 
fulfilment of needs can be 
described* 


OOf/s 


D,A, 




1 ♦01/05 "It can be broken Into A 
phases", 


See 
comments 


0 


Alternative In-- 
terpretatioa 
approach «♦ see 
Options ^ lv*+f-c 

S^'Ct,ion 


1*01/06 "this activity takes place A 
in a nuir^er of fields 
(consisting of individuals 
and/or groups) which have 
specific characteristics" 

- 


See 
comments 


0 


The notion 
"fields" isn't 
relevant unless 
S(L)EA Is 
applied to 
formal evalua™ 
tion. 


1,01/07 These" fields "can and do A 
^ operate independently. 


oo% 


DA 




1.01.08 When they do come into A 
■ contact there is a potential 
for shared activity which 
will depend upon the matching 
activity phase compatablllty , 


002 ttv^el 

See. 


0 


as for 
l,0l//fe. j 


■ 4 
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SMPE^S ABSmWtlOm. PROCESSES Classiflca- 



etc. 



1,09/09 



Critical incident i 
as a cechnlqua It out- 
linaa proceduras for 
collaeting'absarved ' 
incldaQts having 
special aienificanee 
and maetlng systamatl^ 
cally dafinad criteria. 



1,09/10 "Incidant la any obsar 
vabla human activity 
which is sufficiently 
complete In itsalf to 
permit Infarences and 
predictions about 
the person parformlng 
the acts 



1-09/11 



To be "critical'* an 
incident must occur In 
a situation where the 
purpose and intent of 
the act sesTO fairly 
clear to the obsarver 
and its consequences 
are suffielently 
daflnatlva so that 
there Is little doubt 
conceding the effects. 



1.10/12 Phases in the community 
davelopment model. 



1,14/13 



tion Code 



IDT 



Com^tment Scale used 
to assess the amount of 
commitment felt by the 
respondent during various 
critical incidents. 



2.01/14 The SHAPE's system , 
(applies to) "task 
oriented activities'! 



IDT 



IDT 



Reference^ Applica= 
Code tion Code 



001//6 



DA 



DA 



IDT 



IDT 



003 



DA 



H 



See 
cedents 



OQl//b 



H/0 



M 



COM^fflNTS 

' cvbserved * Incidants 
does not always 
apply. 



The commitment notioi 
applies - it was re- 
developed for avalu- 
ation purposas to 
reflect "level of 
involvement" for a 
particular event - 
content analysis 
astablishes if the 
learner uses 1st, 2nd 
or 3rd person in des- 
cribing the event, 

Hodlflad slightly 
to refer to task 
oriented learning, I 
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SHAPE'S ASSU>ffTIONS, PROCESSES Classiflca^ 


Reference 


Appllca^ 


COlOffiNTS 


ate, tion Code 


Code 


tion Code 


2,02/15 -'^y analysis or inter- P 
pretation oi the CD 
activities is adaptable 
to SHAPES providing It 
actually reflects 
reality. 


001/2 


M 


For S(L)EA this 
means "to be 
demonstrated"* 


2.02/16 Wien we assemble IDT 
critical incidents 
together on a time 
sequence , "If necessary 
specific outcomes or 
objectives which are 
achieved can be 
correlated with points 
on the time line? 


OOS 


0 
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Assumptions and Specif ications for S (L) EA 

001/la Human learning can be best defined by the learner. 
001/lb As learning activity includes such non-observable pheno- 
mena as reflecting, meditating, conceptualizing, the only 
way we can assume to validate the learning is through the 
learner's self ■-report of the learning experience. As soon 
as a researcher assumes to interpret observable behaviour 
as the only acceptable evidence for learning, he biases 
and restricts the existence of learning, 

001/2 Any framework we put around a person's learning in order 

to identify it, organize it into conceptual structures and 
evaluate it is subject to the framework developer's own 
biases in learninc * 

001/3 As simple a framework as possible has to be developed in 
order to tap into and then track learning activity. The 
tracking and analyzing of "learner-^percieved" learnings 
activity should be: 

001/4a Accounted for in objective or f actual-inf ormation=f ormat , 
followed by qualitative analysis of the activity f 

001/4b Applied to only those activities indicated by the learner, 
as being significant events in their learning; 

001/4C Convertible to sequential and/or learning pattern (style) 
interpretations * 

Need fulfilment is a global, idealized phrase that is liable 
to vary in terms ofi 
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001/5a How and when needs were identified = some lead to the 

strongest task orientations or objectives at the outset of 
; a learning endeavour; other needs emerge and take priority 

during a learning endeavour. Some people just allow needs 
and, in turn^ objectives to emerge^ others may work with 
both initial and emergent objectives; 
001/5b Fulfilment may turn out to be different in terms of stan- 
dards of performance, pride ^ creativity = before, during 
and after a learning endeavour, 
001/6 Until brain waves and chemical processes are decoded into 
s^jecific human thought patterns and actions, we can be 
totally blind empirically to a multitidua of learning 
activities* We can only, I repeat, only defend the exi^ 
stence of learning through the perceptions of the learner, 
001/7 Significant Learning, There are problems of definition 
; and, ill turn, validation in establishing the legitimacy 

of a "significant event". One could argue that by defini- 
tion, a learning event is "significant" when the learner 
: percieves it to be significant,* 

The factual circumstances identified by learners as des- 
cribing a significant event and the qaalitative data about 
the event brought about by the Q sort component of S(L)EA 
can be analysed. What is ironic^ in a sense, is that 
S(L)EA is an instrument which is continuously applied to 
"validate" significant learning. 



; *This has a Lewis Carrol^ "Alice in Wonderland" tau^ 

tology to it, "Words mean what 1 want them to mean," 

i 
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The Adequacy of S(L)EA to Account for Significant Learning Events 

A set of criteria is required here to enable judgements to 
be made. The events described in S(L)EA log sheets:- 

001/8a Are sufficiently detailed so as to account for at least 3 

of the following I specific time, date, place, other people 
involved, activity occurring at the time; description of 
the outcome or results of the event; an indication of what 
the event either "led from" or was "leading to"* 

001/8b The events described and qualified by the Q sort are veri- 
fiable through either independent observations or an 
alternative assessment instrument. 

Interpretive Option #1 
Specifications and Assumptions Applying to "Phase" Definitions 
002/1 There are clusters of behavioural and attitudinal indicators 
(statements or phrases) which represent either the progress 
or pattern of a person's learning need fulfilment, 
002/2 The term "progress" applies, if the clusters fall into a 
linear sequence, 

002/3 This sequence of clusters as described in SHAPES is called 
"phases" and can be cyclical; each phase may occur over a 
number of significant events, 

002/4 A "pattern" of need fulfilment applies where the learner 

opts for Interpretive Mode #11 and, accordingly, establishes 
his own clusters, labels and arranges them in a pattern 
which represents that individual's learning "mosaic" • 

002/5 In every instance, a cluster would have to be tested for 
mutual exclusivity and the consistency of relationships 
between clusters. 9 
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What Interpretive Frameworks Provide Useful and Meaningful 
Feedback to a Learner Using SCDEA as a Reflection Medium? 

A full scale study of the Interpretive Option #1*** 

is not feasible at this stage. **b Suffice to say, the use of 

123 

the six phase analysis was applied with SHAPES, the Trainer 
Development Program Evaluation and my Practioum experienoe. 
In each instance^ the data generated supported the ocGurrence 
of each phase. This stated with some guardedness because 
the Q sort items identifying each phase are still subject to 
further development and testing for validity as mutually 
exclusive or discrete indicators,**^ 

A short research assignment was carried out to determine 
the usefulness of the card sort items and clusters generated 
for the Interpretive Option #11, This will reported in the 
section following. 

The Interpretive Option #111 (discovery^oriented) has 
not been fully investigated* 



**a _ . 

Described in the introduction of this report. 
Such a study is the very focus of a thesis that I 



am currently working on 

Altshuler, Do«ey and Friedman. 



**4 
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INITIAL INTENT OF THE INVESTIGATION 

Enquiry Into SCDEA's Second Optional 
Interpretive Mode** - Developing and 

Using a Self-Designed Set of SCDEA Card Clusters 

The card sort items used with S(L)EA wera being examined 
for meaningfulness and usefulness. if any interpretive option 
is valid for S(L)EA, it must first be tested In terms of whether 
the user (learner) can -der stand and identify with the terms 
used in the instrument. "Usefulness" of the card sort items 
has a direct bearing on the reliability of S(L)EA. If learners 
cannot work with the sort cards in reflecting upon or evaluating 
a series of learning experiences, then S(L)EA's reliability as 
a measure of significant events is bound to suffer. The major 
indice of "meaningfulness" with this enquiry is the number of 
existing Q sort cards that a sample of respondents accept. The 
major indice of "usefulness" is the respondent ' s application of 
the Q sort cards in creating card clusters and labels. The 
clusters then become the basic framework for usi i Interpretive 
Option #11. The following is a case study rather than a fully 
fledged field survey. No good doing anything until it has been 
pilot tested: 



♦♦Described in the Introduction Section of thin report. 
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ENQUIRY METHOD 



A small lample of Adult Education graduate students?* 
graduates and practitioners/ were asked to sort the S(L)EA 
Q sort cards in two different wayss 
S ort I 

Separating the cards into two piles on a worksheet* 
Pile #1 would comprise only those cards which are 

^) readily understandable - both the following con- 
ditions have to apply l 
i) the words make sense, 
ii) the sentences or phrases are readily comprehended , 

b) meaningful enough that used as items for reflecting 
upon many different learning situations, these cards 
would restrict the reflection process if they were 
left out . 

Pile #2 comprised cards rejected after considering both a) and 
b) * The components and guidelines to S(L)EA were explained to 
everyone participating in this pilot study. In particular, 
respondents were asked to keep in mind that these cards would be 

a) used repeatedly in reviewing learning experiences, and 

b) any cards which were not relevant for a particular 
event would be eliminated. 

Cards placed in Pile #2 were recorded (using the card # codes) 
on a separate sheet of paper and set aside. 
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As a pilot test goup, this seemed appropriate before 
tapping the general population. I'm assuming that if these folks 
could not find meaning and use in S(L)EA^ people outside the field 
would not; it was also anticipated that this group could articulate 
any major technical flaws in the S(L)EA system and r spond as 
"professional " learners , 

12 
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Sort II 

NoWy there is anothar set of guidelines and workeheets, 

fe: Using thm cards remaining in Pile #1, respondents were asked 

fe--- ... 

1^: to again sort the cards but this time any number of piles could 

W-r be created. Respondents were at liberty to use more or less 

h 

than the seven boxes on the work sheet. The main objective in 
carrying out the sort/ this time, was for the laarner to establish 

J h^^/his own clustering of cards according to any common meaning 

that the card statements represent. After this sort was com-- 

f" pleted, the respondents were asked to label the clusters and 

J/. record the card code numbers applying to this label* 

■ . . " 

Respondents were also invited to write up their own card 
items (on blank cards provided)^ as they were sorting the 
existing cards into cluBters* 

■-. The researcher checked with every respondent who used only 

"(■■'■■ '■ _ _ ' _ 

I the seven boxes | if they felt obliged to sort only into seven piles, 
p. A check was made on the total nuntoer of cards respondents 

1^ were prepared to use regularly this time including the items 

: generated by the learners themselves and allowing for any further 

decision arising during the second sorting/ to reject additional 

J card items . 

Research Findings on Interpretive Option #II (see Table #1^ page 14 ) 
Note #1, With even a small sample of people, ii was sufficient to 
idantify patterns and trends in the use of Q sort cards. Cur-- 
rently there are 4 9 sort cards , 4 of which are parallel or 
synonymous statements for testing acquiescence, 

13 
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A|ate.^2. 21 Different cards were rejected. There were only 2 res- 
pondsnts who coincided for 4 of these 21 items. Conceivably 
the frequencies could dramatically change and show out a far 
more revealing pattern with a larger sample. 
tj^fm^^ 3* There was a broad range of responses to adding new 

''customized" cards. Some learners will be quite satisfied with 
what is already in the deck as illustrated by respondent d) . 
It will probably be Important to know in a follow-up study; if 
respondents felt obliged to replace the cards that they had 
rejected as contrasted with adding cards which reflected some 
missing reflective statements that the learner wanted to use. 
Both conditions are illustrated in just these few results; for 
example, respondent e) had no concern to replace the 15 cards 
he rejected?* Respondent c) , on the other hand, seemed to make 
new cards that said much the same thing as the ones she rejected. 
When interviewed she acknowledged this and said she had worded 
them in a way she could accept it. For instance, she rejected 
#3 card (it felt like clickl or uh huhl now I know what I want 
to do) and replaced it with her own card which reads "suddenly 
I knew what I wanted to do," 

4. In an interview held after the card sorts all of the res- 
pondents acknowledged that they didn't feel their choice was 
forced in terms of using only the seven boKes provided^ .As it 
turned out two respondents chose to use less boxes. 



**His pattern of rejection was also different to other 
respondents who did their card eliminations only in the 1st 
sort. In this case, respondent e) dropped 8 cards in the 1st 
sort and 7 in the 2nd, 
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The cluster headings and card frequencies for each cluster 
turned up further interesting facets of the S(L)EA card sort. 
The cards appear to have been useful to the: a because thay all 
worked with them. Here we have a sample of what might be ade- 
quately labelled "Professional Learners", One thing is for 
sure , . . all of them having had grounding in Adult Education 
principles^ practices and corresponding terminology, seemed to 
avoid using terms (or a sequence of concepts) like "needs 
analysis"^ program planning^ program implementation or evaluation. 
However / there may be some more subtle— jargon, patterns which, by 
getting from each respondent their favoured readings, theoretical 
biases, academic orientations, etc, then using a Content Analysis 

Q 

approach^, the conceptual "sets" may be decipherable. 

An emergent and seemingly exciting finding is the "patterning" 
of clusters, if the clusters for each person is plotted then 
we come up with the following two findings as diagraraned pM^ca Iffb} 



Table 1 

Acceptance^ 'dejection. Innovation of Sort Cards 



Note 


Find 


inga 


a) 


b) c) 


^) 


GOMMNTS 


No. of eerds aecepted by each 
respondent. 


42 


43 


43 


50 


33 


Range 33-50 
Average ^43 


Note 


#2: 


No. of cards rejected by each 
reapondent. 


3 


5 


10 


1 


15 


— ^ — 

Range 1 - 15 




#3: 


No* of cards created by each 
reipondent , 


1 


4 


4 


0 


1 




Note 




No, of clusters generated 


7 


7 


7 ^ 


J* 


5 


The placing of 1 
card as a "clusteJ 
was not included. 



Holsti - Content Analysis. 
O 1 i'S 
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SUMMARY OF PILOT STUDY FINDINGS 



Clustfer #B 



Table #2 - Cluster Names and Card Frequencies (cf) 
Respondents 







a 


b 


c 


d 


a 


c 


) 


^'•Contact or Con 
•sTiTiftc'tiQn. fe©tw-» 

AAAA 5 V b W 44 ■ v @ V *f - 

sen ma & the 

^earning 

tplsodeV 


"Initiation 
I to itV 

cfm3 


"A struggle 

^ W >I ^ W 4^ 4>« 5 

^nd mysel f V 
cf-6 


"Feedback" 


"At the \ 
bpffi-Tiiil-Ti^y ' 

I 


■ 


) 


teonfusion- 
Question- 
Introspection V 


''Indecision 
Suspending 
judijernent 'e 


'^Negotiation 
between my- 
self ^?nd 
other',* 

cf=6 


"Skills 
Development y 

cf-6 


"Somewhat \ 
into the i 
processV 


1 

\^ 

\ 
% 

-4=. 


'Hslaxation 
Insight y 


"Sorting 
!Ostabllshlng|i 
the approach» 

of ^3 


"Insight" 

of ^6 


"Cgtfllytlc 
event- 
passive in 
nature*' 

cf^19 


"T*ni iit^rtln^ 
to get y 
some .of it 
sorted out" 

cf^?: 


1 

® 

* 


'Really connect 
Take hold" 


"Focus on 
contents" 

cf^6 


"Sf fialent 
exploration" 

cf=8 


"Catalytic 
event- 
aggreasive 
in nature" 

cf^ia 


"At the end ; 
of the train 
Ing but t 
still with 
the Big 'fray* 
to Come" i 
cf^7 ; 


d) 

1 


'Insight to 
action 

Clarify opera- 
tional pl#n" 


"Focus on 
aelfV 

cf=10 


"Getting Into 
it" 

cf^5 


"Reflection" 

cf^5 


"All in all " 
cf^a 


® 

1 

] 


'Experience 
Implement 
Immersion" 

cf=11 


"Focus on 
Others" 


"Concrete 
creativity" 

cf^7 


pother person 
focus)" 

af»1 




\ 1 

d) 

1 
% 

\ 

f - f 4 


'Reaction 

Digeetion 

Heflection" 

cf=7 


' Concluelon 
What came 
of It all" 

cf=11 


"Avoldung the 
not so ob- 
vious or 
meeting 
myself y^^^ 







ERIC 



16 



Di,«?rflim #1 Individual P'^tterning of l^rd Clusters Cusln.ft Option tj) 



□ b c d e 




Di'^gr^m OtDtion II Clusters "lomTDnred with Option, T ^rnm^sv/ork 



Interpretive Option ,^^1 


Option #11 ^ 


Self Pe^ignf?d Clust^ 


dm 


FraEftswork ■ (LQarning Phasei 




Respondents 








a 


b 


c 


d 


e 


Encountering Interest/ 












Problems 




% 


© © 




© 


Irienti fying N^seds 






© 




© 


Objective Setting 


© 










Planning 


© 


© 




© 


© 


Implement ing 


© 


©®© 


©©© 


© © 


© 


. Reviewing / i^eplnnning 


© 


© 


© 


© 


© 


(Non- sequential Dimensions) 












Smergent 












. Interpersonal 




© • 


© 


© 




Self-directed 1''^ 


© 


© 


© 


© 
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Diagram #1 

Similar to chromosome pateerning, this might be one major 
way of examining learning stylo. The diversity of cluster pat- 
terns was not anticipated, accordingly, the missing research 
step would be to have asked the respondents to place their cluster 
headings onto blank cards and then have them assemble the cards 
in a pattern which represents the relationship of clusters. The 
above diagram is an attempt at Interpreting the relationship of 
cluster headings. There are some more easily identifiable con- 
nections of the card cluster labels provided by respondents, for 
example 

Respondent a) ' s cluster headings seemed pretty straight- 
forward as a linear progression. 

Respondent b) forking at ® leading to®,® &® 
I on an equal level, lends itself through the common term "focus". 
I I am assuming olU8teri®)^(^ and are happening either simul- 

taneously or alternately and not sequentially. This junction of 

join with ® as a logiaal outaome for all three 
dimsnsions, A similar assumption was made for Respondent c ) 
but this may be a relatively faulty phase assembly, not so much 
£ or clustera(T)^ @ & ® as 1* would for clusttre @ ,@ ^ 

I Respondent d ) seems to have set up a phase sequence with 

I branching at (J) and becoming alternative operational "modes" 

4 
P 

I rather than phases. In addition, the somewhat disconnected 
I looking cluster (that is the"other person focus") suggests 

the respDndent Identified a Nonsequsatlal Clustbr. This means that 
we predict the occurrence of any card items belonging to that 
cluster to appear at any time during the learning sequence. 

18 
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Respondent e) g like a) , has an easily decipherable linear 
sequence but oddly enough the respondent seems to miss a cluster 
heading for what he refers to as the "BIG DAY", (Broken oirale 
( D; , has bean included to suggest where this phase might appear,) 
Beoause the data was aseembled^ analysed and reflected upon well 
after the respondent's card sort^ it was not possible to check 
into the need for a "re--sort" or establish that the omission of 
\Dj was intentional* 

Diagram #2 

Comparison of the respondents' cluster labels with the 
Interpretive Option #1 (Learning Phases Model) ^ shows that all 
respondents identified one cluster which is labelled with words 
that match a "review-^cum-^ref lection" period. Comparable res^ 
pondent labels at the upper end of the learning phase sequence 
are less discernable* In at least 3 instances a combining of 
the Option #1 first and second phases seems to be occurring 
(encountering inheres t (s) /problem (s) ) * respondents clearly 

indicated a planning phase (^^for Respondent a) and f or 
Respondent b) ) * 

All respondents identified an "action" or implementation 
phase. Comparison of Ofi^^^^^^^^^^ with respondents ' ^^^^^^ is 
an important extension of what was first pursued as the area of 
inquiry* If Option"^ I has any merits, it must be because learners 
can approximate the relationship between the cards that they put 
into clusters when using Option #11. Those people, incidently^ 
identified the interpersonal cluster accurately and two created 
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olustars that approKimate the "self-dire-cted" dimension used 
in Option #!• 

The final test to this entire exercise is finding out if 
the learners can consistently realise their clusters and the 
coinciding patterns through the learning eKperiences they haveg 
(S{L)1A Card Sort Data)* 
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The Evaluation Research ApplicatlQns of S(L)EA 

Program EvaluatlQn conventionally traces the outcomes of 
learning objectives/ eKamines how well instructional components 
(materials # audio^visual aids^ instructors^ etc*) facilitate the 
attainment of the objectives and attitudinal facets such as 
participant satisfaction with the program (or sub^components of 
it)^ meeting learner eKpectationSf follow-up needs of partici- 
pants/ etc* Apart from the attitudinal measurement (eKpectations / 

needS/ etc) S(L)EA can derive data for the other aspects of 

■J . , 

Program Development* However/ thi^ occurs in very different 
and searching ways* To start with/ learning objectives can be 
appraised through S(L)EA in terms of a) how well they were based 
upon the needs identified/ b)how they are linked to planning/ 
c) how much conunitment/ follow--through and review the learner 
had/ and d) finally/ where other objectives emerged which out- 
competed or led to the dropping of original objectives. Whatever 
instructional components had a significant role to play in facill-^ 
tating these learning objectives are. not obtrusively tested 
through using S(L)EA, That Lm, no part of this instrument asks 
the learner what he/she thinks of the instructor/ or say the 
programmed instruction materials/ film or group discussion. If 
any such phenomena impacted upon the individual/ then they would 
be relayed as part of a significant learning event* 

S(L)EA is best applied to evaluating programs which have 
a high level of learner participation/ including the needs 
identification/ objective setting and planning phases of program 
development* Involvement in the entire evaluation process is also 
an obvious requirement when using S(L)EA* 
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EKperience with S(L)EA in a Program Evaluation 

In the appendix to this paper there are some extracts 
from a Program Evaluation Report, This assignment used S(L)EA 
to make a ^jiimber of different Interpretations about the program 
(Trainer Development Workshop) that was evaluated. Of important 
note here (in si^mary) is that the criteria of a "significant 
event" (see 001/08 a^b.c, above) were largely met by the 114 
S(L)EA log sheets turned in by IS respondents i 

- All of the events described on the log sheets^ met at least 
3 of the criteria set out in 001/8a above ^ with the highest 
frequency of qualifying factors being 'time^ activity' 
occurring at the time and outcome ' or result. The latter 
factor was subject to value statements as much as objec- 
tive or factual statements. 

- 87*7% of the S(L)EA records qualify as Smallest Meaningful 
Units (001/8C) • 

A Spot check using the terms of reference suggested in 
001/8b was carried out with a few of the workBhop participants 
via telephone Interviews i the B(L)EA log sheets handed in by 
workshop .resource people was cross-^validated through Indepth 
interviews and program documentation assembled by the resource 
people (see iKhlbits of pages 11 and 12 of the Trainer Development 
Workshop Evaluation Report in the Appendix) . 
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C, METHODOLOGY ^ DATA COLLECTION 

. a) Individual Level - Performance and Process 

Data on the Individual learner level was collected from 
four major sources: 

1) Participants In the Trainer Development Workshop com 
pleted a pre-course self-diagnosis form aksing them 
to identify 5-10 priorities to be dealt with on the 



and high) and then to establish a priority for their 
own learning needs* 

The priorities set by participants v;ere then used by 
the evaluators to establish what the Individual's 
learning uojectlves were for Phase I, 

2) Further data on individual learner objectives^ were 
collected from the learning contracts developed by 
each participant for his or her own learning In 
Phfse II. Each contract contained tht Indivi dual's 
objectives for this phase along with a statement of 
an action plan for meeting these objectives* 

3) An attempt v/as made to assess each participant's 
significant learning events from the program, at the 
same time avoiding the problems inherent 1n asking 
people to directly link their learning obj^Aiyes to 
their learning outcome^s* It was felt that a question 
naire or other instrument that asked participants to 
be accountable for what tney set as priorities and 
goals would become a 'self-fulfilling prophesy* of 
objective achievement* 

AlsOt we as evaluators i«re concerned to find out if 
the program was successful with or without learner 
objectives and to allow for emergent or unintended 




categories* Participants were asked to rate them- 
selves in terms of their competency (low, average. 
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outcomes in keepiny v/ith the design of the workshop. 
It becanie an almost impossible task to 
measure the degree of achieviment for each Individual 
learning objective. This was for two reasons at 
least - the first being the mere quantity of objec- 
tives ^ and the second being that most ware not stated 
In miasurable tenns. 

Accordingly 5 we adapted from the Shared Process Evalu- 
ation System (1974 - Dr. L, Davie et al . - see Bibli- 
ography) an instrument which we called Sign1 ficant 
(Learning) Event Analysis (S(L)EA). Basically, the 
instrument provides data on what a participant per- 
ceives to be his or her most significant learning 
events during the course of the workshop. The instru- 
ment is fully explained by the kit that was made up 
and sent to each workshop participant (See Appendix E), 

4) Finallyj data was also collected on individual learn- 
ers from the post Phase I evaluation form developed 
, by the resource people (See Appendix F). 

Program Level - Performance and Process 

The essential purpose here was to gather data on the pro- 
gram goals and the program events. Five major sources 
were utiliiedr 

1) An open-ended Interview was conducted ,,ith each of 
the workshop resource people. An attempt was made to 
focus on 

- their perception of the program objectives and the 
program's rationale^ 

- thei r perception of the actual program events j 

- any surprising or unexpected events and/or out- 
comes J 

- any judgements they had made about the program's 
performance or process ) 

(See Appendix G for the Interview guide). 
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2) Data was gathered on program objectives and events 
from a variety of paper materlalSj i*e*i workshop 
advertisement. Bureau publ1 cations , handout materials ^ 
etc, 

3) Each program resource person also completed the Sig- 
nificant (Learning) Evant Analysn's Instrument, They 
were asked to identify what they saw as critical events 
In the program. Thus, they completed the instrument 
from a different perspective than the course partic- 
ipants , 

4) Also, data from the S(L)EA's, completed by participants, 
was used to make some judgements about the accomplish- 
ment of the program objectives anU 'about the relative 
significance of various components of the program, 

5) Finally* data was also collected on the program from 
the post Phase I evaluation,, develop jd and administered 
by the resource people* 

c) Program Level - Efficiency 

The primary objective of this analysis was to compare in- 
puts into the program.- the relative amount of time 
devoted to each of the events in Phase I; aggregate 
expenditures for Phases I and II - with the outputs - 
significant learnings. We were trying to discover the 
most effective input of resources relative to the amount 
and variety of learnings as they occurred in the Trainer 
Development Workshop- 

The secondary objective was to analyze the funding pro- 
cedures for the program and to investigate alternate ways 
of achieving similar results. This objective suggested 
an approach that would allow us to find out whether a 
representative control group of qualified Trainer Develop- 
ment Workshop applicants who did not participate in the 
program^were able to find alternate ways of meeting their 
original learning objectives. These alternate methods 
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would then be compared to those employed by the Trainer 
Development Workshop, A questionnaire was sent to 
ten eligible applicants. 

Funding procedures were analyzed separately under 'Pro- 
gram Costing's Including incomes and expenditures for 
the Trainer Development Workshop, 
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D) METHODOLOGY - DATA ANALYSIS 

Taking into account the conceptual background in Section A, 
the concerns expressed in Section B and the research instru= 
ments available to us, our plans for analysing the data 
emerged as follows: 

(i) All objectives (at both Individual and Program levels 
for both Phases I and II) were rated^with reHability 
checking by at least one other evaluation team member. 
The rating was in two categorieSs "measurable" or 
"non-measurable". An objective was measurable when 
Its formulation was*. 

a) specific or relating to only one variable j 

b) describing terminal behaviour characteristics j 

c) including standards of attainment (such as time 
limits)) 

d) discreet in terms of fitting into either the cog- 
nitive or affective domains (psychomotor skills*- 
not appl 1 cable) . 

Any combination of at least two of these criteria 

were considered acceptable. Table 1 presents these 

results. 

(1i) Each participant's Significant (Learning) Event Analysis 
(average of 8 per participant) was rated (again, with 
reliability checking by at least one other evaluation 
team member) as follo^-/^: 

a) Either a "Global" recall of a significant event or 
a "Smallest Meaningful Unit" (S.M.U.), S,M.U. 
was applied where the description of the event 
pertained to half a da;* (or three hours) or less 
durations and 

b) Level of involvement in three categories , as fol= 
lows: 

First Person - where the respondent recalled an 
event where he/she was acting in a trainer role* 
making a self-initiated or 'proactive' inter- 
vention or expressing a trainer competency^ 
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Second Person - recall of an event where the 
: respondent's main role was a reactive one (e*g,, 

: reflections of some learning experience = lecture, 

group discussion, learning materials, etc.) 
Third Person - recall of an event where the respond- 
ent was acting as an observer of other people* s 
behaviour (without making either an Intervention 
or being in any way a recipient of whatever the 
observed parties were doing); this event was 
usually described In highly judgemental or evalu- 
ative terms. 

Refer to Table 1 for the results of both ratings 
in a) and b) above. 

(111) All of the Individual's learning objectives and 
significant learning events were rated (and rel1= 
ability tested by at least one other evaluator) to 
be either Role orientated (R,0.) or Personal 
Orientated (P.O*). Role orientation applied to 
events and objectives relating to a trainer com- 
petencyj expression or practice of some trainer 
skilU interventions or even reflection upon theory 
or observed data (e.g., andra^ogy group 
dynamics* etc*)* Personal orientation pertained to 
objectives or events about either intra or incer 
personal matters, which had no bearing on the per- 
; son's professional development as a trainer (but 

was significant to that Individual in terms of 
personal growth). Results of this analysis a^e in 
Table 1. 

(iv) A number of comparisons were made with the data. 
: Figure 1 sunmarizes these comparisons. 

t 



Figure 1, Comparisons made with data 
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Notes: (a) to what extent do individual objectives 
match program objectives? (Table 3) 

(b) how many learning objectives relate to 
significant learning events (Table 2a) and^ of those 
objectives that match, how many significant learn- 
ing events relate to them, that ISj vie re intended 
outcomes? (Table 2b) 

(c) what bearing do the significant learn- 
ing events have on the actual program events? 
(Table 5a) 

(d) how do program objectives relate to 
actua, program events? (Table 5b) 

(e) what bearing do program objectives have 
upon the individual's significant learning events? 
(Table 4) 

(f) how do the indivlduaVs learning objec- 
tives compare with actual program events? 

(Comparison not made) 

Efficiency of the Workshop (Phases I and 11), that 
is, the cost of the program In relation to outcomes 
or cost per significant learning event and per 
learner (Tables 6a and 6b , 7 and 8). 
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RESULTS AND DISCUSSION 



Individual Level (Performance and Procegs) 



TABLE # 1 - Classification of the learners' 
objectives and the learner's SCL)EAs for 
PJases I and II. 
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LEGEM) : OBJS - Objectives ; SCL)a\ = Significant CLearning) Event 
Analysis ; N - Number of participants; Ph I or II = Phases 
1 and II of the Trainer Development Workshop; S,M<U* = Smallest 
Meaningful UnitC 3 hours or less); R,0. ^ Role Orientation 
P*0, ^ Personal (Intra/ inter) Orientation] N/A ^ Not Applicable, 

Table 1 shows that thtre were a large number of individual learning 
objectives In both Phase I and IT, The average number of objec- 
; tlves per participant dropped from 14 (Phase I) to 8,6 (Phase II), 

i There may be at least three explanations for thisi 

i) the objectives for Phase I were largely generated by the 
\ competency rating form provided by the resource people before 

1= the workshop. This form provided an overwhelming amount of 

\ Information to be handled effectively in the program an 

I Idea that was supported through Interviews with the resource 
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i1) while the form asked for five to ten priorities the range 
shown 1h Table 1 suggests that In fact some Individuals far 
exceeded the maximum in both phases 

ill) The development of objectives for Phase II occurred at the 
end of Phase I and it would seem that the experience of that 
Phase assisted Individuals in focusing more clearlx on objectives 
to be worked on in Phase II, 

While the numoer of objectives decreased, examination of the 
data shows that the vast majority (69.8%) of the individuals' 
learning objectives were not stated in measurable terms. These 
findings suggest that participants* objectives did little more 
than provide a general orientation. In this regard, Table 1 also 
shows that the bulk of individual learning objectives (85,5%) 
received an RO rating suggesting that individuals were largely 
aiming at Improving or gaining trainer competencies. Percentage 
wise, there was a 101 increase in Personal Growth orientation 
during Phase II which reflects upon some participants* reorienta- 
tion and the manner in which they intended to express their 
learnings from Phase I. 

Table 1 shows Individual learners identified 114 significant 
learning events, with an Increase of one learning event per 
learner from Phase I to Phase II, It is difficult to know 
whether this is an Indication of more learning in Phase II or rathe 
the fact that the Instrument was completed towards the end of 
Phase II. 

Most significant learning events were SMU*s giving some reliability 
to the identification of significant learning events In terms of 
a short period of time (half a day or less). Also, about half 
(52%) of the SLEA*s were in the first person Indicating that 
learners were relating a majority of their significant learning 
events in terms of events where they played a major role as a 
trainer or program planner. 



TABLE #SA I Comparison of SCL)E\s and 
actual Program Events, for Phases I fi II. 
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